Image and video, when they are transmitted or stored, are always represented in compressed form because of their huge size. Block-based transform coding is the most widely used compression method for images. The order-8 DCT is first employed to compact signal energy and then a quantization process is applied on the transform coefficients. Block-based transform coding is also widely used for video data which will first be compressed using intra/inter-frame prediction to further compact the signal energy. Then quantization is performed on the residual transform coefficients of video signal. Such video coding method is often called hybrid video coding for its use of both transform and predictive coding. Hybrid video coding is adopted by most video coding standards including H.264/AVC. In both image and video coding systems, quantization is the main source causing the coding artifacts such as blocking and ringing artifacts.
In order to reduce artifacts in hybrid video coding, there are two main approaches in integrating deblocking filters into video codecs. Deblocking filters can be used as postprocessing filters or loop filters. Postprocessing filters only operate on the display buffer outside the coding loop, and thus are not normative in the standardization process. Loop filters operate in the coding loop and filtered frames are used as reference for subsequent coded frames. Loop filters which are used in both encoding and decoding operations need to be specified in the standardization process.
FIG. 1 illustrates a block diagram of the encoder of H.264/AVC. As shown, an input video frame Fn is presented for encoding and is processed in units of macroblock. Fn is compared with a reference frame, for example, the previous encoded frame Fn-1′ by the a motion estimation 100, and then a motion compensated prediction P is generated through the conventional processes of a motion compensation unit 200 and an intra prediction unit 300.
P is subtracted from the current macroblock to produce a residual block that is transformed by a transform unit 400 and then quantized by the quantization unit 500. The quantized transform coefficients are entropy coded by the entropy encoder 600. The quantized residual coefficients are obtained after inverse quantization by the unit 700. Then the coefficients are inversely transformed by the inverse transformation unit 800 to produce a quantized residual block. The motion compensated prediction P is added to the quantized residual block to create a reconstructed block. A loop filter 900 is applied to reduce the effects of blocking distortion and the reconstructed reference frame is then created from a series of reconstructed blocks.
In general, better performance can be obtained using loop filter than postprocessing filter.
Early loop filters were designed using enhancement-based approaches to reduce coding artifacts. They applied low-pass filters near block boundaries. In the art, a deblocking loop filter (DLF) was proposed and adopted by H.264/AVC standard. It operates by performing an analysis of the samples around a block boundary and adapts filtering strength of low-pass filters at the boundary of each block. However, these enhancement based techniques do not result significant subjective visual quality or objective Peak Signal-To-Noise Ratio (PSNR) improvements.
Restoration-based techniques were also proposed to suppress the quantization noise optimally. The optimal linear Wiener filter is a well-known technique to suppress noises. It was proposed to reduce quantization noise in video as postprocessing filters or loop filters. Significant objective quality improvement can be achieved by the restoration-based techniques. Wiener filtering requires knowledge of auto-correlation and cross-correlation of original signal and noise, so filter coefficients need to be transmitted in the bit stream of a video coding scheme. On the decoder side, the filter coefficients are extracted from the bit stream to construct the optimal Wiener filter. As a significant number of bits are required for the representation of filter coefficients, so one optimal Adaptive Loop Filter (ALF) is used for one frame. The ALFs can reduce the average distortion of each frame globally, but it cannot adapt to different local regions within a frame.
A variety of adaptive methods to consider different local regions within a frame were developed. Block Matching 3-Dimensional (BM3D) method uses Wiener filter thresholding to restore 3-D Discrete Cosine Transform (DCT) coefficients grouped by block matching and has obtained excellent performance on image denoising. In “Sparsity-based Deartifacting Filtering in Video Compression” of Jun Xu, Yunfei Zheng, Peng Yin, Joel sole, Cristina Gomila and Dapeng Wu, IEEE Int. Conf Image Process (ICIP2009), Cairo, Egypt, November 2009, it shows that BM3D method only works well for intra coded frames as a postprocessing tool. An effective way to use BM3D method as loop filters both for intra and inter coded frames has not been found.
Block-based adaptive methods were also proposed. A representative method is the quadtree-based adaptive loop filter is described by T. Chujoh, N. Wada, and G. Yasuda, in “Quadtree-based adaptive loop filter”, ITU-T SG16 Contribution, C181, Geneva, January 2009, and is one of the coding efficiency improvement tools in the Key Technical Area (KTA) software. Quadtree-based adaptive loop filter (QALF) allows a block filtered or not filtered and represents the side information using a quadtree. Further improvement is provided by another tool in the KTA software, which is called quadtree-based adaptive loop filter with deblocking loop filter (QALF+DLF). The scheme allows a block to be filtered by ALF or DLF. The side information is also represented using a quadtree. The scheme achieves state-of-the-art performance both on objective and visual quality improvement. However, the block-based method does not fully capture local statistics of an inhomogeneous frame.
Non-local Kuan's (NLK) filter is designed based on minimum mean square error (MMSE) restoration. It is proposed by Renqi Zhang, Wanli Ouyang and Wai-kuen Cham in “Image Postprocessing by Non-local Kuan's Filter”, Journal of Visual Communication and Image Representation, Elsevier, accepted for publication. The filter works by capturing local self-similarity property in a picture. Based on the NLK filter, two image postprocessing methods, DNLK and OCDNLK filters, are proposed. Both the two methods use dual layer filtering process. The OCDNLK filter combines the advantage of the overcomplete transform and the DNLK filter. They all achieve the state-of-the-art performance on image postprocessing.